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ABSTRACT , . - • ^ ' ' \- 

The prbjecVs goal was to enhancer i;^he outride ^ 
evaluation of the Opportunities industrialization Centea^^. Career 
Intern Program c4XP) Mi-tli supplementary, Griteri'pn-^ref ^s^nc 
assessment measures iden€;ified through a litersiture review and an 
^analysis of CIP^s goals. The review (presented separately) . referred/ 
to ovey lOO programs of dropout prevention and ybcatipnaX development 
for the disadvan;taged; in reports haviTtg an evaluation component,'';^ 
^^essment measures , have been ,identified. Described in the documentor 
these are not limited to crit^ripnrreferenc^?)!^ techhigt^esir which were 
infrequently fotitnd# tteasures/tended |tpVbe obvious and .'already ^ 
included in tlie CIP package^ Unique measures are discussed in the 
dpcumeivt within .the framewptk of a generalized evaluation, model, . 
constructed to pi^ce./^^^^ data from literature in perspective and to 
determine the' reguifements for.. additional aissessment measUreis^ In^ 
general^ the document cphclude^B^/ the measures fpund in ^'the llter?^|:ure 
were related to sjjecif ic, ishort-tera program e.xpectatlpns, bas^d on 
iimisdiate ^program putcome^i,, and net empiricallj'. tiedM^o ultimate, ! , 
iotig'rrange goals; , the data preset^ incpmj^ete. inA fJ^agmehted 
.pictures of .prbg^rajns* progresjs and success, ft new approach is 
suggested^ which asks not whether the program obj^.ctives ha^ye beerf - 
joet, but whether they iiefe. tli^ correct objectives; *«What have we 

for this individual?" >i«^•>^/^t•/r.T^ / - 
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CHAPTER I . • 



' INTRODUCTION 



The approach to evaluation has been modified and refined over the past 
several' years* Even the basic concept has changed from one of assessing 
"program success" to that of a process which is totally integrated Into the pro-^ 
gram Itself and attempts to contribute to "program success/'. It is no longer 
sufficient for the evaluator to arrive ^at the end of a project and, using his tradi- 
tional measurement tools and statistical tectmlqties, to assess the success or 
failute of the program. This unitary, after-the-fact approach has in most cases 
produce4 quite urisajtisf actory iresults which were useless .to both tl^^ose jAio needed 
to make management decisions about ^ the program and to those who were involved in the 
progtam development. ' In Jiis opening remarks to an invited seminar on evaluation, the 
late Dr/ S. Rains. Wallace expressed his concern that meaningless evaluation was in, 
fact rtiining the cutting edge of innovation* He stated also that^ "Perhaps most 
Importantly, many of us share an uneasy but firm conviction that our evaluation 
l>roblcms camiot be solved with the trad;l tional scientific methods, and paradigms 
and the statistical refinements we have developed for their analyslb^t^d^nter- 

pretatit)n" (American Institutes for Research, .1970, p. vi) . . ^ . 

' ' ' ' - ' ' , \i 

It would appear that the current emphasis on indtovation and, social. change 

has prompted a new look af evaluation. There is an increasing need to be able to * 

demonstrate that programs or parts of programs are indeed effective orxat 

^ieast that they can be made effective. There is a need for those implementing 

a program to he able to^der^ive feedback from what they are doing so that' changes 



fean he made in a timely and effective manner before the progrto endg.. There 
is certainly a need fpr those in management positions to be able to know if 
progress is being made or if a, program has achieved the desired results.. Thus, 
the technology of innovation has brought with it a parallel development in the . 
technology of evaluation. An attempt is being ^ade to deal with the problems" of 
assessment in a sophisticated but practical manner so that the^^ results are both 



valid nnd' useful. 



outcomes. 



A nevr^pcabulary lias developed. Terms such as objectives, 

r 

long rimge goals are used to specify the desired results of a project, while 
terms like "process evaluation," "product evaluation," "formative evaluation," 
and "summatiye evaluation" '^re used to describe the targeted ^areas^ for. the 
Vario.us components of the overall evaluation program. ^ The concept of linking. 

a measure or instrument to a meanit^gful reference point, group or situation 

\ . ■ ^ 

(criterion- reference) has received recent support for"* assessing ^ange. Thfere, 



is continuing research on how best to examine the real ultimate impact of a 
new program or innovation. This entire evaluation fichnology effort is embedded 
in the problems of workirfg in the "real world," Reality* brings with It the 
problems of limitations in dollars and time as well as various practical, social 
and political barriers to implementing the desired program and/or evaluation 
plans, * i • 

Qur increasing level of knowledge and awareness^^ both, innovation and ^ 
evaluation had^ resulted In a more comprehen^ve integrated approach to both. 
In addition, the identification of objectives not only aids the evaluation 
effort, but it also requires that those who are implementing the program 
define their goals more precisely. The data from the formative and process 
evaluations can be fed back to those responsible fpr the prdgram so/that 



changes can be' made in the methods or resources applied.' Therefore, b*oth 
the innovation and , evaluation programs are tied together to their, mutual, 
benefit/ EvaliiationThas been broadened to ihcl^xde an assessment of tjie on- 
going .processes and components of the progfam, and sharpened to include the 
measurement of the degree of success in attaining certain precise -objective^ 



The gaal of this project was to enhance Gibboney and Msociates* ^ 
existing evaluation of the OIC Career Intern Program by the identif icejtion 
and recommendation of supplementary crit,erion-r^fe*renced measures. These - 
recommended measures wotild be aimed at assessing both immediate and long- 
term program goals. The procedure for identifying these measures were: 
C A. A revlew'uf existing literature in the area or drop-out ^ ' 

prevention, career intern programs, and "other related areas. 
B. An analysis of the objectiyps and g^als of th€i OtC Career. 

Intern Program. . ' v - , 

It was Intended that the recoinmended set of measures be consistent with 
the established evaluation and objectives, be supplemental rather than redundati^t 
to measures already included in the evaluation program, and be sensitive to CIP 
program , outcomes. 

Tlie results of t;he literature review are presented in a separate report 

by Tiedeman and Miller-Tiedeman entitled "Career Initiation im Association with 

Alienation frbm Secondary Schools: An Operational Model and Its Literature,'* 

(ERIC/CICE Information Analysis 5A) . This survey of 'the literature contains 

Qver 100 references to related programs in drop-out prevention and the pyepara- 

tioij of the disadvantaged for' the world of work through vocational training, 

# 

educational enrichment, and counseling. It highlights the different approaches 

• " " ' A . " ' 
taken and discusses the results of some of the more significant prdj ectg^^ 



, Using .the references identified in the literature /eview, each report 

was examined carefully to determine if any type of evaluation was conducted 

' ^ / ' ' 

ds p art of tjie study ♦-^ For thdse ^projects having an;6valuation component, 

they were agairfi^ reviewed to identify the specif ic^bbjjectiyes , and the measures 

usiad to asses.^ the program outcomes. It should pe noted that the number of 

programs employing lde,ntlf Lable measures werg^ donsiderably less than the 

' ^ ^ ' . ^ • ' . \ J^' ' ' 

100 ^rorcrgrictcs In tiu» literature review. // 

... — gr - ' . 

Because of the limited data availablje from related programs the measures 
' n 
described in this doitumeot are not lim:^|:e4 to criterion-referenced techniques 

as originally intended but include the.^ntire spectrum of measures used in the. 

' , /'V . • i 

various, programs . It was hoped tha|J^^iip^ a broadening of the scope "would 
increase t|ie utility of t,his docum^l;^t. For -the most part, the measures 

■ ' ^ ' /// ' : * 

tend, to be the obvious ones and are already accounted for or included in the 



Gibboney evaluatioji package, ^nrijere were some unique measures identified and 
t^ljeso will bo discussed In /he /following sections within the framework of a 
gonoralizod ovalnntlon numi:}./ - ^ . ^ 

The literature .its^f,'' as typical of most of the so,cial science literature, 
tended to be quite variable with regard to b^th quality of the research an^ 



quality of the reporting. Only a limited i)umber of the projects reported in the 



literature attempted any type of objective evaluation. Very few of thede 

/ '. - , / ' 

studies precisely described their assessment effort; and /fewer described,' to ^ 

any useful level*, 0f detail, the characteristics of their objectives., measures, 



- / 

subject populations. In the small number of cases where the descriptions were 



adequate^ the subject populations tended to be so limited in si2e and scope,^ and 



iim 



the measures so pr^dgram specific, that the results could ^t best be described 

as inconclusive dnd any attempt at generalization made with, only the greates^t 

of taiftlon. T^erfefore, despjife the ^over lOt) references tD related p^pramff* ."^ 

in the literature review^ objective data on various kinds of measures could . 

only be abstracted from th^ 20 studies cited in this reports * • 
/ i . * ' * * 

^ As a /final 116 te regarding the literature in this area, it should be 

recogni^d that most of the studies of interest were reported tn the so- 

called "fugitive" literature. Similar information about related programs and 

stud/es was sought bbt generally not located in formal archival journal 

litijniture. 

•/ 
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. • In order to place jtlje data from thei literature In perspective and to 



determine t^te cequiretn^ryts for additional assessment measujreg, it was 
f necessary tp cori^.truct a ^generalized model of the OIC/CIP Program .evaluation. 
TJie moded' :p resented here is not necessarily congruent witji tl>e pne that 
,undci;'M«3 the current evaluation effort. . It was , developed rinly to serve as 



a ftpmc of reference for this following discussion. 

After site visits>v.,discussipns witH* program sfcafl and the reyiew of a.. 



considerable amount pf program materials, the following* six. objectives were. 



identified as the desired major program outcomes. To;^ assist the students in:, 
"^.l.^' Obraining ^academic credential's (dip,loma <Jr ^GBD) ^ ^ ' 



C 2. Making an dpprbp^irdate career choice^ .V-. . / f ; ' / -^r ; 

3.,' Having a variety of ^ practical "h§*nds-6n" work experience. y 
' 4>/.. Oeveloping a sef of relevant basic, academic sUills. / 

' . 5. Acquiring a set of social.^skills required' for job success.'. ^ 

, • 6. ^'^ImproVlngx'their sejf-image and reducing^' al^ ' 

iqphs evaluation ef f Jrt^:]^ ^iejjed as having four (4) phases/ The interaction 
, pf :l^he progr^am o,ytjS£me$ , and the variotis^je^Lalua. tlon phases ^jejrepresented in ^ 

. y Figure 1.- . Possible ineasures are also shown for each 6utco!ne..^^hase I is a 
, ; Formative or ^Process Eyaluatibn phase for assessing program progressT^^-his^ 
, is conducted while the progr^un i^ ongoing, and is aimed at measuring the progress 
"being made in various components.^ Such information can be fed back into the ^ 



program to feff eet improveraents. The second and tjiird phases are' Summat^ivl or i 

•- -. • ' ' ■ , - .■ ...... - ■ ; ' , 

Product Evaluations. Phase II is directed at "Immediate Program Outcomes" while;: 
Phase III examines the "Post Program Objectives" or longer range .program 
effects. The Wiase II outcomes are Che eiid points of * the. Phase I progress, 
evalirntfon, i.iv, it is keyed to the same general outcomes. The object of 

w - . ' ' ^ \ ' ' . ' ' ■ r ^ ' ; 

Phase 41 la tlu* evaluation of the, effect Ivencvss oi the program oi;^_,tlie program 
•elefiRMits^ while Pliasc U I' looks at the brpadqr questions of program scope — ^ 

.Is the program ^loLng tlie, right things? Is it doing enough*? 

- *' . ' • • • - ^ * 

^ It, is expected that by, the Phase." I^I evaluation the program participants, 
have: • ^ ' ' 

1. Completed the program and have' received their high school 

' . ^^ . ' ^ ' 

^ diploma or GEDi \ • 

2.. Made an appropriate choice of x:ar^ers which is mature, realistic^ 
and achieveable, and based upon useful knowledge ^hout tliat career 
vgaLnod*Ln the program. ~^ 

'3. Participated ia a variety of practical experiences in various . 



• A. Attained an acceptable level of achievement in such basic 
, academic skills as reading and math. . 

5. /Recognized and developed the necessary social skills required 
' fpr successful employment. ^ . ' \ - t /* 

6. Developed a good self-image, sel^-confidence and r^dufced feelings 
♦ of alienatibh from society *and its institutions. 
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All of these, outcomes feed "into those for the ^hase'ilIi|evaluatloni" * 
Phase -III Isi conducted some time after the end of-, the program iii a ttonth^ 
six months or a year. Obviously it can k'lso conducted several times at 
various time intervals afte^ the end ot the-prdgfan. Ij^^^ddressek^.the Wo 
major isxpected i^utc'omes: . - ' - - w ^. 



X'. - . « 



1. Is the program participant ein|51oyed;/s.contintiing , his" education 
or training^, or a combination of the. two?- 

2. lioes'the program participant;;iave a positive 'attitude toward . 
himself an^^s 'community? \: 

' ^ The measures here can bfe 'som^ of the same dnes usid.in Phase I and 
II; however, some new ones. may. have to be inf^oduced* It may be necessary^ 
to examijie surrogfate indicators of the desirfed outcome^. For%xample', , 
geographic mobility may be an. indicator of/self-confidence .-^hd deWased 
s^ienation. Promotions .on the ■ jo^^^h'ighfer earnings than equivalent -program 
/ 'hQn-^^rticipants, and ' participation in cbmmuni'ty' activities may ail*be ' 
Indicator^sof the expected program outcpmes. *; • 

. Finally, ?h^e IV Iqoks at. the Lon&fRaiige Goal^of the prpgram and 

, attempts to assess ^it^ltimpeHb^aj^^ The.qu§st^ion addressed a#. this 
point (some time in tiie fbt^ure) is no^^^j^th^r prc%ram objectives, were - ' 
achieved, but whether or not thev JWer e "Me \dght^ objectives The ultimate 
criterion for program suctess must 1^. to determ^^^e if the individual paz:ti- 
' ^^cipant,^is ie'tter ,ofr for having been in\^ie Ffrpgrain>\.Thia, obviously, * ' 
capnotfbe done until long after , the program ^nds7 althou^ some- Interim 
^ ""^PJ^^ress^ltieasures are certainly possible and desii^le, ^ 



At«{^.lii» Huch Murrogiite measutfoK ys voter'" registration,, comrauhlty partl- 
cXpat ion, .continued education. and assessment of life style may. bq^ indica- 

. . /x ^ ^ ^ . ' ' - ' \ ' ' 

tLve of -:tJie expected long range goals of: • > 

1. Productive 'and satisfying employment*, ' . . / 
^Z. ' Participative ^itlzensHlp:*. 

3. Gbjitinued selfr^impri^ . * . ' . ' . 

These^tliree gS^ls .all contrljtute to the individual's ' ' eng agemen t'^ wl th 
societ^^(Seeman, 1972).* Thky are difficult to measure- because they are. , 



removed froii^ the basic progi^ outcomes in terms of both time and lever of 

' ■ "X ^ ' 

ab55tractness« ^wever, one possible appro^icl^ to neasureraent i? *to. assess 

-r^^ '"-•^ ^ >v ' ' > . / . ■ ' . z 

engagement ^by Idokihg at the investment the indiv::dual is willing to make , 

in htraself ^'and/or his "^cbmuntty, Stich investment may be in terms of time,' 

effart and money (Krug and Jong, 1974) . The Inve? tmeht may ie^ a way of 

quantifying commitment and thusNj^ngagement." UnJ ortunately , .there is little 

in-^the literature which reflects onsthis phase of evaluation. If measures 

* , are to Be used in Ph^se. IV, then ^n jeffoj^t will have to be undertaken to 

7^>aevelop arid validate them.. . / 



The^jterm "engagement" is used here by Seeman as an antonym to alienation 




\ 



•CHAPTER 4. 



'THE MEASURES 

. \ 



The measures .used to evaluate the related programs identified In the 
literature rah the gamut of conceivable assessment tools. They included 
measures of IQ, aptitude, achievement, personality, attitude, and interests. 
In addition^ such program related measures as attendance measures, grade-ppint 
average, rating scales and interview' forms were used. A complete list of 
tliose measures identified in the literature is presented in Appendix A by 
category and article referencing the particular meastire. Virtually all of 
the measures fall into the Phase I and Phase II evaluation categories, i.e.', 
they are measures of progress and immediate progranj outcomes.. Most of them 
were giyen at .the Beginning, during and at the end of the various programs! ' 
Very few measures for Phase III (Post Program Obj.ectives) and Phase IV 
(Long Range Goals) were, ^entified^- ^ * " \ 

Only a small number (11 to be exact) of the related studies identified 
Included any type 'of comprehensive evaluation effort. Those that did take 
a broader approach to evaluation tended to be similar to each other in ths^t 
they attempted to as3ess the- same general areas of, progress and immediate^ 
program outcomes. The program objectives most frequently examined were the 
assessment of: * I 

1. Progran effectiveness in. overcoming deficits in academic skills, 
particularly; reading and-. arithmetic. ^ . 

2. Personality and attitude factprs associated with improving the 
part:icipant*s self-iraagfe. , ' 



3., Changes in feelings of ali-epatioii. from the mairi stream of ^society 
" and its institutions; e.s.> sc^ipol and the world of worki. 
4. Th6 acquisition of certain social skills nece^sparir for holding a . ^ • 

: job. . - •. . . ■. ■ . 

'5. The; acquisition of technical skills needed for obtaining and 

\hplding a job. ; " . ' - . . ' . , ^ 

> . " * • , 

Similarifty between the goals and objectives of the OIC/CIP .prograj is 

obvious. It should bd noted, however, that there is a major jdifferenfie 

between the OIC/CIP program and other progVams identified* This difference 

lies in the combined career education /work experience approach 't^ea in the CIP 

' « ' ' ' ' / / ' ' • 

program. While other programs did include pers6nal and career couflSseling . 

as p^rt of their multi-faceted ap[prbach to the drop-out p^JToblem, we were 

unable to identify ano.ther project which had^ a duaj: e^jphasp^f work experience 

with- career education and guidance. Thus, there was little un the literature 

regarding the assessment of the maturity and level of kx/owledge with which 

' - . ' * ' ' ■ " . . ■ / . ^ 

"tarcJer decisibns are made* l^^Ax^ ' 



A. Previous^ Comprehensive Evaluation Efforts 
Described below are a series .of studies whi^ approach, the ^valuation 

• ' • ^' ' / ' . V 

problem in a comprehensive nianner. Tljese evaluation efforts are described 
and the measures identified. In most cases, the, measures were used as pre- 
and post-program tests. The actuat results associated. with these measures, 

if any were reported , will found in the f6;Llowing section on "Results ,^ 

" ' V • \ * ' . ' ■ - . ^ 

from Previously Used Measures .^^ ^ / - . ' ^ 

. Hamburger .(1965) reporting on a program designed to aid disadvantaged 

youth with a co-op work study program reported on data collected from 108 
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"Co-op 'students- and 91 Controls. The data were collected In 1962763* Hl^ - 
^measures -included:. - / . 

1. Stanford Achievement Tests.,- Advanced 'Battkry^jPorm- JM,^ 
' Reasoning and Rearfing Subtests. 

btis Intelligence , Tests, Self-Adminiktered^.^ainma Form FMi This 

h ' - ' ^ ' . ... 

test was selected because of its similarity to the typical Civil 
Service exarainatioif. • - ^ ■ - \ 

3. The Differential Aptitude Tes : Battery, Form A, Clerical Test. 
This test was selected because it was especially relevant to the' 

/ 

/ large number of cooperative students who entered the office work 

^ area. 7 . . 

4. The Work fnventories developec by Donald E. Super and Associates.. 
This- measure was ujf^d to assess changes in: work attitudeis and values 

5. The Life-Planning' Questionnaire developed by kartin Hamburger. 
Th±8 measure was us^d to compare and contrast levels of aspirlition 
and levels of expectation in the vocational and. educational areas*. 

6. The Incomplete Sentence Blank 'developed by U.V. Rotter, High 
School farm. This instrument was used as a personality screening 
device. 

<N , - , • 

7. Job ratings vere obtained from each of the cooperative students. 

8. Interview data using ai structured interview schedule were obtained 

(I 

on each of .the students.' ' . !^ • 

9. The Revised Occupational Scale for measuring socio-economic, status,. 



developed by Hamburger, vas used to rate levels of occupational 
aspiration and expectations. 
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10. Additional data on the students wer6 obtained from school records, 
^ fIrcHo iticludl»d absences , latenorfs, disc tpling, slips, grades, and 
school tests. 

Leubling and Trobe (1965) reported on a program* \Jhich focused on the 

vocational^ educational, and personal problems Of school ^drop-outs between 

the ages of 16 and 18. Their evaluation program consisted of the following: 

* 

1. Wechsler^inteliigence Scale " , 
. 2. Monroe Silent Jleading Test ' . ' ' 

^ 3. Woody-McCall Aritbmfe tic. Fundamentals . , , i^" 

4. Minnesota Cle^rical Test 

5.1* Vocational AdvLsory Service Simple Posting Test (Program Specific) 
(). Minne55ota Spatial Relations Test 

O'Connor Wiggley Bloclc _ * 

• 8.- O'Cdnpor ^Finger and Tweezer Dexterity ^ 

9. Minnesota Rate of Manipulation' (placing and turning test^ 

10. Vocatiofial Advisory. Service "Information Schedule" (a projective^ 
' questionnaire ^developed for the project) - 

The following tests were also used on a selective basis: 
l; The Thematic Apperception Test- 

2. Bender-Gestalt Test - - 

3.. Draw-,a-person Test . y * ' 

11. Porteus Mazes, 

5. Kohs Bloclc Design • 

6. Stenquist Mechanical Assembly ' 



A "7. lER Girl' s.tMecUanical . Assembly Test 

8. ' Pennsylvania Bi-Manual Test 

9. ' Vocational ^dvisory Spelling Test 
10.. Gates Reading Test - * 
ll^ Peabody Picture Vocabulary 

-^12'.- Otis Arithmetic Reasoning 
Of the 263 clients, 218 were administered a full battery' (158 were male 
and 60 were fcmaio). However, the test battery was modified when necessary 
to coni5ist only of non-verbal tests since many of the clients had language ^ 
aad reading deficiencies. , In addition, many of the clients had negative 
feelings about anything resembling an academic pencil and paper test so that . 
only performance tests yere used under those pircumstances as well. ' 

In "A Pilot ProjecL to Develop a Program of Occupationat Training, fpr 
School Alienated Youth" (1967),^ a comprehensive evaluation program was attempted. 
It was, however, limitei to the assessment of progress and immediate outcomes. 
The experimental design [included the use of a control" group so that unwanted 
extraneous effects that result from normal growth ah,d the effects of factors 
alhc^ than the program could be considered In the analysis. However, it 
was impossib*le to obtain a r^andom sample of control students and the sample 
size was, limited to fifteen. Therefore,, comparisons between the experimental 
and control groups had fo *be made with extreme caution. Data' analysis of the 
experimental group itself .showed ^N's ranging from 7 to 39. 
V The measures used in the evaluation are listed below: 

^feasure . - Characteristics or Variables 

^ 1. California Study Method Survey School adjustment, study habits., skills 
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* 2. California Test of Personality 
3. rPAT Anxiety Scale 

* /i. I.orge-Thomdikc Intelligence ' 
5. Urown-Carlsen Listening 

Comprehension Test 



6. 
7. 



4 . . 

Personal and social adjustment. 
Anxiety, level* " ) 
Academic aptitude. • 
Ability to comprehend ideas arid 
^to remember sjLgnifilcant details - 
of 'listening. ^ 
Understanding general science.'^ 
Understanding general mathematics. 
Understanding social studies* 
Reading vocabulary and cbmprehension 
Occupational interest.* 

Clarity in reading and conversation. 
Work attitude in occupational ski^lls. . ' 
Self-concept, social behavi'br/ atti-- 
- ^ tude toward society, school adjustment, 

and work attitude* - 

It was planned that except for the case study and thg occupation rating scale, 

all Instruments were to have been admiijistered prior to entering the programed 

upon leaving. However, the original test battery proved to be too long and cumbersome 

and the battery was reduced from 13 items to the seven instruments marked with an 

asterisk in the above list.- 

^ Kaufman, et al*, (1968a) conducted a study to.' examine the Relative effects 
! ' ■ ' * ' ' ' 

of skill training versus a diploma ^program for a group of high school drop-outs* 

There were 60 students in the Diploma Group, 28 in the Skill ^Ttaining Group, and^ 
the Control measures, were obtained f rom ^'0 to 63 students with actual group 



STEP Science Test 
STEP Math Tesfr^ 
8. ST.EP Social Sctudies Test 

* 9. California Reading Test . 
10. Minnesota Vocational Interest 

^ Inventory ^ 

* 11. NYU Speaking Test 

* 12. Occupational Reading Scale 

* L3. Case Study 
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size .depending on the' test. The measures used to evaluate the effects of- the 



program were: 



Measures 



.-1. OTIS Quick Scoring M^ntaL Ability 



Characteristics or, Variables ' 
A tt^^dttional verbal IQ test. 



'.Test, GAMMA Test Fdnns EM and FM. 



2. -Tlevised BETA Examination 



3. Culture Fair Intelligence Tests 
(Forms A and B) 

4. -Edwards Personal Preference 

Schedule 

5. The Cough Adjective Checklist 



A non-verbal intelligence test 
corisisting of various performance' 
V * subtests. 

I 

A npn-verbal intelligence test 

w - 
l' using relajbionships between 



6. Coopersmith, Self-Esteem 
Inventory 



7< Minnesota Vocational Interests 
Inventory * ; 

8. Stanford Achievement Tests 
(Advanced paragraph meaning 
test Forms W and X; Advanced 
Arithmetic tests W and X) 



I geometric figures. * 
Measures of 15 personality variables. 

\ / . 

A personality measure by which 

\ 

Ian Individual describes himself 
using any* of 300 common adjectives. 
A measure of self-resteem in which 



the respondent checks off whether 
the various statements ^described 
are like or unlike himself. 
Comparison of rest>ondents interests 
to those of current members of 
various occupations. . 
Standardized achievement tests in 
reading and arithmetic. 



^ Kaufman used the •standardized measures to assess immiaiate. program effects 
on intelligence, personality^ vocational interests, and^'^ academic achievement in 
reading and arithmetic.. However, no long .term follow-up was made. 

Erickson and Hamler (1972) reported on the results of the Benjamin 
Franklin's Urban League Street Acadenqr which was set*,up to deal with school 
alienated youth. The major objective was to .help drop-outs to return to 
school oir^^to enter the world of work prepared and ^tivated to adiust and 
achieve satisfactorily." Iq order to do this, thefA^adeny was prepared to * 
provide help in: • / ' \ j^-* 

^ Overcoming academic and vocational weaknesses; to enable the drop- 
outs to continue their education. . . 



^ Overcoming alienation ;from self and others ,f|rom the educational 

setting, and frpm school personnel. v '-^ v 

^ Remediation of below level academic skill^M 

^ Developing, and acting upon, self-confident' It titudes particularly 
by participating in continued education* \ I 

^ Developing positive attitudes toward self, school, and society. 

o A ' 

Developing a mutual trust, communication skiljs, and legitimate 



concern for their community^ ^ 

■ - i ^ ' 

The evaluation measures used were: % ■ 

' 'ii.' 

Measures Characterist ics orlvariables 

1^ School records, interviews with .School ap^, emplojrment status, 
school personnel, parents and 
cominuftlty / ' 



2. * Stanford Achievement Test 
3* Questionnaire Data 

4. Actions to continue education 

5. Unstructured Interview with 
community, students, parents 
And staff ' 

6. ' .School records 



Acl>ievement in English anxl rnarth* ' 
Measure of alienation. ' ^ \ 

• / ' . ^ ; . 

> Development arid application of 
/ self-confideibce. 

Development of trust, communication, 
^and concern for the academic program 

and the community. , ^ • 

Cost per student. 



^ ; A unique feature, of this particular evaluation was the inclusion of measures 
of self-development. The reduction of alienation and development of self- 
confidence are items that were used in other evaluations. ' However, the attempt 
to assess the ability to act on that self-confidence to. continue one^s education, 
and the attempt to measure the development of trust,j -communication, and concern 
with the community are measures un4.ique to this partijcular study. 

Another approach for comprehensive evaluation wks presented in a study 
by Joseph and Almen (1970) of a Work Opportuiifty Center (WOC) . The WOC .program 
offorod training in a variety of skills (e.g., machine shop, graphic arts, 
-f^icyJ^ preparation, etc.) combined with courses to help develop communications, 
social studies, and*' mathematics skills. ^ In 'addition, the student, was given 
an opportunity to earn high school credits at the WOC. Although several 
standardized and specially developed tests -and measures were utilized*, most of 
them were used for diagnosis and prescription rather t^ian evaluat;^on* Educa- - 
tional history, reading level, and family. back'ground were some qf the types 
of data gathered on each^ student. However, some pre- and post-test data were * 
examined to determine the effectiveness of th6 program in terras of imiriedlate y 
outcomes. These measures inclucjed attitude an4 self-perception qLu^tionrialres 



developed for the .program, an analysis of attendance ^ patterns, and a record 

" ^ ' , ^'^ " ' , y ^ ' 

of * the stttdeii!;s*v P^'?^-^^^ future.,^ The follow-up studies to assess the- 

' ' . " -'^ . * ^ ^ . . 

more: long" term effects^^ included: f 

1. -job sucpess as measured by employer and ^tiident.evaluation^v 

2. Academic success as measured by the ^^-entry* of the students to ^a 
- r. » ' regular hlgb' school program. 

3. "Ihe vaiue of tlie^ training as measure 
the graduates were dping^ six months 

Data were collected on^ the nianber atifd types of jobs secured. 
Drane (1971) examined a wide range of viriaBles when assessing Project 
STAY. ^This project, conducted in St /Louis, Missouri, consisted :of a broad 
approach to the treatment of drop-outs. The program consisted .of work-study , 
, guidance, continued education, ^social and personal .adjustment, cutriculom 
. revision, and after schpol activity components. The project objectives, and 
measurement techniques are listed below: ./ 



d by a determination of what" 
and one year' after leaving WOC 



o 



Measures 

1. S<ihool records 

2. DEMOS Drop-obt Scale. 

3. School records 
Af School , J^ecbtds ' 

5. Questionnaire and social 

•* worker visits. 



Qbjettives ? ^ 

Reduce the- percentage of drop-outs. 
Improve the attitude of-^drop-outs 
or potential drop-^outs toward school. 
Reduce, absenteeism- in school. , 
Reduce tlie number of suspensions in - 
school. 

Entry of students into post-high 
school educational programs and/pr * - 
successful post-'high. school employment. 



StuHerilt dpntinuation In school- / * 
after pregnancy # 

_^ Rediice ta.te pf ^ failing grades. . 

t-- , , ' . ' ' 

4 ' ' ' ' 

- t - . 0 ' 

Improve achievement in haa±c sTcills.^ 



6v School records 

7. School' irecotds 
♦ &. The Iowa Test of Basic 

Skills for, language and ^ ' ' o " « 

' " ^ • c • . . \ . 

Arithmetic • * - ' ; 

if* ^ 

9. Sdiool records' ; Have students participate meaningfully 

in afte,r school activities.""^- 

Using subjects from two Neighborhood Youth Corps out-of-school projects,, 
Freeberg and Reilly (1972) attempted to develop and validate^kn evaluation test 
battery. The measures used and their characteristics are 3ii^ted below: 

S'- . ■ ■ ■ 

c MeasuTP. • 

1. ' P*]jfactical Reasoning Tests--simple Zip. coding and mai^ reading tasks. 

2. Job Kn^SB^e Test — a multiple choice test on a variety of jobs. It 
icluded questions on educational requirements,, starting, salary, ^ ' 

primafyMtasks performed, houi^s of work, etc.- ^ ^ \ ' ' \ 

3. llob-S'eeking Skills Test—a multiple choice test dealing, with ways 
of lookiilg 'for jobs.. . - 

1/ ' . ' ' " ' ' • . 

4. , JbbrHolding Skills. Test — a paper and pencil test in, which the 

' ' , • ' ' . ' ' ' ' ' '\ 

respondents imist describe the appropriate behavior called fpt in 

described situations. The items j?9|l designed .to eflcit responses 

concerning appropriate dfe8S,Vpunctuality, etc; ' ; - ' 

5. Vocational Aspiration Minus Plans TestV*^ measure of the discrepancy. 

' ' ' " . . ^ ' ' • "I;" ' ' ' _ ^ ^ ' ' ' 

. ^ between vocational aspirations and vocational plans i. . • ' \ 



; - , .U. AttltiuloK ScnleiT—to nssesstttfc I tmfcs^ toward authority-, self -^esteem, 

■ motlViiLton Tor vocational adiieveraeiit, ;and willingness to accept 

• / V -deferred gratification/ ' ' - " • 

• — . ' r - ' . • t ' - ^ ■ • - ^ - : 

7. Vocational ^Qjt'eresjts Scales — to measure the respondent's preferences 
I. * , , fx>r performing various kinds of tasks, suidx'^ its clerical, servfc^, 

- i % M .technical, e%c. . . . . ^ 

• f .8. Standardized Achievement Test (Paragraph meaning siA-t^t of the 

^%anford Achievement Testi, Form W, Intermediate Ij Arithmetic and j 
. , Vocabulary ^sub-tests from'tlie Adult Basic Learning ^Exam. Level II, ! 

; . ' Form A; a Figure Analogy iTest/ A Maze- Following f est; and a Hote 

Memory Xest.) 

/ ' The measures described above jwere compared with criterion measures i^jj,^;^ 
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order to determine their predictlA^e validity. The criterion measures" were 

divided into t:wo pa'ttjs; (1) program completion' criteria and (2) post'-prpgi^am: 

criteria. The initial. 'cMteria fjjr program completion consisted of adjust- 

ment to the training program, social adjustment Cincluding adjustment to -the 

commtnity and the family, police coiftact, peer Gratings)', and job aspiration 

levt»l. Tlie last criterion considered salsgcy expep tat ions,' the level of the 

current job and the level of^ong' range plans.; The post-program criteria 

consisted of general job success and "adjustment as determined from employer 

proficierfcy ratings, financial sayings,^ ability to retain employment, length. 

o^ 'stky on' job,, fioiiy adjijstmentV and police contact. In addition, some 

Assessment of 'striving for vocational success was mi^de by examining job , 

• ' ' ^- . \ /- ■ • ' — . ''\ ' , - . , . ' 

promotion, amount; of first raise, problems, with people in the community, 

' ' ' ' ' * ^ ■/ 

the quality of the next job des ired, and thia ease with whi ch he was contacted 
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by the ijtt^^rvlewe*. Arid' finally, tJie subject's ''stability-mobility'' was 
toxnmlried by dt^terinlnlng the number of pancef?;7ii^ineerviewed for a job,, the 
numlHif of jobs head, the ,vf sits to state emp;io)^^ serviMv and the number 
bf peaces lived.. Tlie- preliminary data were collected and factor analy2ed. 
Tlier^\^e promising results on some measures. ..However, the small sample 
proliibitedS^awing any significa^^ '. " 

In-evaluat^g the drop-out prevention program designed- to work -witJuninth 
grade students ln\leveland public schools, Kilbane and Fleming (1972) 
conducted an assessme^^of the degree to which the program was able to achieve 
its objectives. The lmiae^4.ate program outcomes and measures are listed below: 



Immediate Program dutcomes 
J. A majority (75%) of the students 
. would nttairt w "uomal" rate ^ 
_ of progress in reading while 
in the project. 

2. A majority (75%) of students 

* * 

would attain a "normal" rate 
of progress in mathematics 
while the>prpject. 

3 . Students: would' aJttain better 
school marlcs while in the 

^ * ' ' . r': ■'• r ^ \\ 

^ program than "fthey did prior 
.to particip^£i&t;* - \ 

.4. Studeints would eviden^^^ a higher/ 
attendance rMe,: while in the^ ! ^ 



Measure * ' 

Comprehensive test of basic skills 
in**vocabulary and re.adin^ comprq-r 
hensioni 

Comprehensive tests of basic skills 
computfa^on test. 



"Grade-*point average. 



School record^. 



-^3 



schopl, and t^eachere. 



5. PrograiuNstudents vbuld evidence School records • 
« lower dropr-out rate thaa other . 
ninth grcide students In the^ r 

project scliooli^v ' j * ^ 

^ I \ - r * • . V ' 

6» Student^ would ilnprbve^^he ' Teachers * reports 

. quaility 6f their wrij t^n 

o^Lasswork' and' hbmewoik assignments* 

'7* ^Students' attitudes, toward self ,^ Attitude sbcales. 

would improve* 

\ ■ - ' ' 



Projept Outreach (l9Ti\ was a broad school-wide program to reduce drop- 



outs* The immediate program autcbmes and evaluation instruments used ^re 



listed below: 



. Immediatgi Progt^am. Oiit^cbmes 
1. Enrollment in school would 
< , maintained during the fira,t:' 
^ . project S'ear by 90 of the 100 

*s^udents Identifijpd. as potential 

drop-^outs. ' ^ \ 

• , ' ' ^ i( 

2* Percentage of D and F grades 

of •the targeted students would 

be decreased by 10%* 

3* Absenteeism of ' the targeted 

Students, would be decreased.-^ 



Measure 



Qfjficial school records* 



Official school records. 
Official school^ recpjrdfif.- * 



Iowa Test of Education Develop- 
ment and the Iowa Teat of Basic 
Skills. ; ' ' ' * 

Attitude Survey (Evaluator Design 

' \ ' ' 'J 

Semantic Differential jcype) 



A, keadihg compreltenslon mid 

vociiftulafy wou ld be. inerci.wod . 
by five (5) per pentilie' points 
for pargeted students* 
5. The attitude of targeted stu- 
dents toward adult authority 
and toward their environment 
would be improved. 

■■ . ' I - ■ ■ 

Hprnbosjtel, et al|», (no date) conducted a /complex and comprehensive 
evaluation on a program to assess the effects bf various kinds of training 
program^ on\the rehabilitation of the drop-oqc. ,Eour groups of drop-*outs were 
Fol lowed "pver n twa year pje^iod. There were three experimjsntal groups :^ one 
received vocatioivil and academic training, .the second received yocational 
training^ alone , ^nd'^the tl^ird was given acad^ic training alione. | The jEpurth 
was a conerol group aiM received no training^. In addition, a fi^th (second, 
control) group was comp^ed of, students who had started but did riot complete 
any of the experimental pfograms.. Betause of att^-ition, the two i control 

• ■ - ' -f A ■ \ • ■ -y \ ■ ■ " 

groups w^re later combined into orie. There were 118 subiects in! the Academic 
Vocational Group, . 115 in the Vo&atidnal Group, 5.9 in the Acaiiemlc Group, 
and 46 in Control Group* I. fUnfor^na^ely, as time went on and attrition took 
plape,, all qf the groups shrank. in s^e.^ The 'N's. for the. v^arious groups at 
the thrie testing points aire shown lrf\^bie I. During thfr follow-up , the N^s 
ratiged ^'rom 4 to ^0* : f r ' ^ ^ \ . . ' ' 

The. resfearch design, consisted of: ' \ . 
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PRE-TEST — the assessment of differences among the group prior to 
troatracut* , ^ . 

BKllAVlOU DURING TRAINING— this included measurements o£, the following: 
Achievement * ' /. » , . 

2. Social adjustment - . , ' 

3. Interests 

4. Anxiety 

5* Social class orientation 

6. Rural-Urban orientation 
^ POST-TEST — the assessment of differences amori^" the groups ^ftet draining. 
^ BEHAVIOR AFTER TRAINING-- the assessment of clients using the following 
crfttirla: 

1. Social adjustment ^ 

2. Interests 

♦ 

3. Anxiety . j 
4V Aspirations 

5. Job placement history 5"- , 

6. Job success ^ - 
- • 7. Job satisfaction 

^ ^ TWO-YEAR FOLLOW l|P— an assessment of thWdifferences among the groups 
\ two years after the training program • 



The progress ar^ immediate outcomes of primary concern were: 
. 1,- ^ Academic achievement \ • . ' 

2» Changesji^ characteristics 
3'. Achievement in vocational training 



ill .-uUiiLluu, the long range assessment 'criteria were related to h\e graduate's : 



•■•t. Ability to obtain, and hoM a job -.\ 
* 2. AbJl'Ilty to perform the job satisfactorily 
- 3. Continuation of education and/or training 
4. ^General satisfaction vith life 



\ 



The evaluation instruments and administration schedule are shown in Table II. 

The General Aptitude Test Battery yields 9 aptitude test scores and Is 

widely used by the United States Entployment Service. The Sequential Test 

it . 

of Educational Progress measures academic achievement in reading, writing, 
i;oclal studies,^ mathematics/ and science. The California Test of Personality 
was used to measure "life adjustment" and is a co;mposite of the "personal 
adjustment" and "social adjustment" scales. The Kuder Preference Record, 
Personal Form A, was used to measure interest iu social situations related to 
work. The IPAT Anxiety Scale was used to measure clinical anxiety ih>^ 
objective and standardized manner. The Social CJLass Inventory, developed by 
S« .Sutker, is an expeirimental iitstrumeut and is intended to reflect either 
.middle or lower class orientation. , The Rural Orientation Inventory is also 
an experimental Instrument authored by S. Sutker. 

Other key data collection instruments consisted of the Initial Data 
Sheet which was develpped to obtain basic personal background data, the School 
Drop-out Ii;iterview Schedule used to acquire demographic and background informa-^ 
tlon about^.each subject, and the Youth Opportunity Survey which was administered 
at six month intervals after gaining. In addition, the Bray fi^^drRo the < 
Job Satisfaction questionnaire and the Goetzel Job Performance Sca^e were used 
to measure a subject's vocational success as viewed by both themselv^d and 
their employers. ' - , . ' 



<5 \ 
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■" TABI^ II - • ■ "■ - 

I 

ln$truraent Admlnl«tr;atipn Sclifedule 
(lIorhbpHtel, e£ alO 



Instrument , 



General Aptitude Ig^est Battery , 

Sequential Test of i Educational 
*Pr ogress' ^ , , 

California Test of Personalit)r ^ 

IPAT^ Anxiety Sc^le Questionnarie 

Social Class^Xai^Orientatioii 
'* Inventory(S. Sutfcer) 

Kudpr Preference Record & Persona 

Initial Data SJieet Record 

School Drop-out Research Interview 
Schedule 

Every Pupil Scholarship Test in 
^3rpewriting ' 

Hiett Simplified Shorthand Test 

Mullenbruch Office Skills Achieve- 
ment Test / ' 

Office Machine Mechanic Test 

Purdue Trade Information Test in 
Welding / 



Pre-Test 



During 
Treatment 



X 
X 



X 
X 



4 



X 
X 



Purdue Test 'for Machinists and 
^Machine Operato;rs 

Cosmotology ^est 

Mullenbruch/Garage Mechanic Test 



Teacher^ s Rating of Vocational 
Skill Performance 



tith Opportunity B^ollow-Up. Survey* / 
.tzel Job PiBrfotraance Scale* 
O Bray ]field-Ro the Job Satisfaction Blank* 



* tftilisged in' folloir-up study 
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.ALthough.the small number of subjects tends to lay open to question 
some of the findings of -this evaluation study, ;Lt was undpubtedly one of 
the most ambitious and well-planned of any of ^ the evaluation programs located 
in the literature* It included various assessment techniques prior tO' 
training, during the course of training, and at varying periods following ^ 
training. In addition, the measures covered aptitude, achievement, personality, 
and values, as well as specific skills required for particular grades and jobs* 
.These were supplemented by structured interviews and ratings scales as described* 

♦ B. Results from Previously Used Measures 

The data in this section are presented in the general order of the 
model described in Chap£er 3* They range from measures af progress and immediate 
outcomes to post program objectives and long range goals. It should be- noted 
that quantitative data were not available for all of the measured previously 
mentioned. ^ - . , ' ' 

1. Measures of Progress and Immediate Outcomes j^, 

a. Continuing Education-*-- Retention in School ! - ' 

Chance and Sarthory f oiind^ that out of 64 enrollees in their program, 56 
remained and 2 joined another program^CTob-Cijrps) . "This is a retention 
rate of 91% which equals the retention rate for the junior and senior 
high schools at large despite the fact that project enrollees were ^ 

selected 3pecifically because they were considered most like to drop out***" 

• ' ' ' " i ' 

r . , 

(Chance and Sarthory, 1972, *p* 213)*. ^. * : ' , . 

Sarthory (197^ reported,^^that of the 71 potential drop-outs in a .work 
experience program 26*6% did not complete the.year*"" This compares with a 
national drop-out ',rate of over 30% and over 40% in some urban school districts* 



In i^yo'jec't STAY conducted in the St. Louis Public Schools the drop-out 
'• . " ; ■ . ■ I 

rate was reduced from 17.S6-% to 8.97Z at one high school and from 1.46% to 

0% at a' second over a two-year period. .In addition, 96,! 1% of the*knowi¥^ 
pregnant girls (^r-100) at one of the high, schools continued in school. '' 
Suspensions at the two high schools, which had a .combined enrollment of over 
3500, were reduced from 3.36% to .38% over the same tWo-year period (Drane, 
1971). ■ ' . 

. 4 ■ 

Kllbiine and Fleming (J972) reported a drop-out rate of 4.5%' for their 
program as compared, with 4.7% for all other_9th grade students in the five 

V 



project sdiools. 



\ 



Project Outreich (1973) reported that of the t^tal students enrolled 
in an Enriched Education, program <N=215) only 5% dropped out as compared 
Iroo-c 



with, a 7% overall di 
b > Continuing 



I 



:op-out rate for the school. 
Education — Attendance ' 



Davis .(1S72) reported that absences declined (N«15) from 306 to 50 
from the first^to second semester.; Chance and Sarthory (1972) fouad a 
decrease in aV^tage absences from 46.6 to- 40.9 days after one year of the 
program (N=?56|^i 

Hamburgel?? ^oiihd improved attendance for a group of co-op students 

(N-108) when c^pared ^th a contrql grou^ (N=9l). Average absences fell 

• ^ , ' \ ' 

from 8,09 to 6pX)9^for t 

the controls. Rd^ston 



one year of 



le co-op students- andN^nt from 7.10 to. 7.31 for 
C1970) reported a 39% increase in attendance after 



a sp4<Sial program to keep potential drop-outs in school. 



Attendance Increased from 85*8% to 88,4% for the boys and* from 84. 4Z , 

to '86, 3% for girls in the. Production Workshop Program for 9th' grades. 

The average for attendance for all 9th grades in the regional school pro- 

grdm was_85.6% (Kilbane and Fleming,. 1972) , while in Project Outreach ^ 

♦ 

(1$73) absenteeism dropped from 11% to 9%. . ^ , • 

Drane (1971) found no significant ^improvement in attendance for parti- 
cipants in the STAY (drop-out prevention) program, .ilowever^ at one school 
the percent of student^ absent 30 days or more dropped fron XO.4% to 
.' C. Intellectual Achievement 

.In 'aistudy of students in a cooperative work-study prDgram reported ^ 
by Hamburger in ^1965, it was found that co-op students (N=108) h^d a mean 
Increase of 5.50 points in IQ vs. * 1.55 points for ths contr^l^^^^^l) . 
There was no significant difference between the g?:pups prior the pro- 
gram (Co-op X = 84.0; control X = 85.17). The IQ test used ya^^.^^^^Q^s ^ 
Gamma , ^ Form EM. ' \ ^ 

\ In a program desig^d to give a group of droj^-outs the acaaimit^ 



ijrepWration necessary to qualify'^for a diploma, Kaufman, et: al. , (^3.968a) 



repojrted a statistically significant increase in average IQ for the'experi- 
jiental igroup (N=60) . Verbal IQ was raised from 92.1 -to 94.0 (Qfcis Quick 
ScirLng Mental Ability Test, Gamma Tedt FdrmS EM and FM) . . " ' 

d.^ Basic Academic Skills — Achievement Tests 

Hamburger (1965) found no significant increase in the Stanford Achieve 
merit test scores fpr Wprd Meaning and Paragraph Meaning in a saniple of 

■ J^. ' ■' ■ ' 

^08 co-op students vs. 91 conpcol students. He meas'ured the students' pre- 



and post-progr^ participation. He also found ^no• significant increase- in 
arithmetic ;abil:^tY as measured by the Stanford Achievement Tests as a result 
of partlclpatioti In the Cooperative Work .Study Programs. ^The co-op studentq 



werg compared with 



gain 



NQwYork (1967) re 



a group of controls (N=jl) who also sliowed no significant 

i 



The Public Se^rvice Career Program pSCP) of the City University of' 



[)orted .across-the-board increases in the California 
Achievement Test (CAT). The PSCP program is aimed at assisting past high * 
^cho^^^ individuals to prepare for the high school equivalency examination; 
The CAT grade equivalence rose from 7.8 to 8.0 for reading, 7,2 to 8.4 for ^. 
arithmetic, and 7.6 to 8.9 for language in only four months./ A g?:ade of . -. ' 
•9.5 i^considered the "equivalency" level, and appeared to ,1j'e in >each'^of the/ 
9-12;mpnth program.,- The Gates Reading Survey was used as a screening device 
but .one recommendation 6f Che progress report was to switch to the Adult Basi 
Leam^hg Examination (ABtEV. It was ^ stated that' the AkjLE'ls un.timed,' is" not ' 
aS' culturally biased, and /appears to be mor^ precise. 

Kaufman^ et aT., (196^3), reporting on an academically" oriented d[rpp-6ut 
prevention program, found a ^^ISN^n reading and aiiithmetic achievement measures 
asjjneasiij^d by the Stanfprd, Achievement Tests. .The AdVanced P^ragtfapTi Meaning- 
and Advance;^, Arithmetjl^ests, Forms W ^nd X, We uped, \ Vj ' 

Efrahe (1971) r|ported na significant improvementi 'lii -the' ^ Test pf 
a|ic skills ,Ci?nguage, arithmetic, vocattilafy. and reaQlng) for projedt ^ 
TAY partl,cipantg.. • ' . . ' ' 



In the <:ieveJana Public ^Schcrbls Production Workshop Program for potential, 
drqp-outs Lii the 9 th^ grade ^ Kiibane and Fleming (1972) reported that 71Z ^ . 
of the participants *'(N =2 3) demonstrated "normal" progress in Vocabulary and. 
56% "normal" progression Comprehension. This despite the fact that marked 
deficit^ in these two areas wer^^easured in pre-test scores (grade 4.4 in. 
-^VBtabuliHcy. and 3.8 in Con(prehetfsion) . WTes^' changes and the Computation 
'test, changes are shown ^low. ' " ^ 
Boys ^ ' ' ' • Girls 



Vocab.\ . Comp. Computation . Vocab. Comp. Computation 
Pre-test 4.0 .3.5 5.3 4.7. • 4.0 5.9 

' ■ * . >, ' 

Po8L-te.st 5.7 5.0 6.0 5.8' 4.9 ' 6.0 ' 

' - < ' I 

Using tlfc Stanf^prd Achiewmetit Tests, Erickson and Hamler (1972), ! 

.repotted increas^es in both reading . and mathematics for^ students in a . 

"Sftrefet Academy." The^i,ncrease was measured for 20 students over a. seven 

^' ' ^ ' ^' - ..." ' - - 

-month. pei;;jLod. English levels went .froih^"6 yrs. 4 mos. |o 7 yrs. 1 mo^ 

while Tn,a*thematics Increased f rom 5 yrs. 8 mos. to 6 yrs. 2 mos. 

Using^the Iowa tes$|fe of sEducational Development and the Iowa Tests , 
of Basic Skills, 1^ no jecc butt*each- (1973) reported that the mean percentfie 
for Rcadinyg^Cpmpr^hensi0n from 10 to 23 (Nf*'127). T^ese werj^aggregate 

.scores f or pifogram participants, grades 7 through 12.- No increase Was noted, 
h0MveTy for mathematics skills. ' , • ' ' 

In a program for school alienated' youth in Norwalk, ConiiaStticu using 
;13 test, variables, only i:he vocabulary section of the California Reading^Test 
showed a signifie^ant Increase over the coritrals.- The experimental group 
improved fjrom 7.4' to 8.3 in grade placement vs. 8. 7 tp 8.8 for the controls. 



All were^ln the ftCh grade. Of particular Interest^ *lt was additionally 
'fouuil that there were 'stgu If leant correlations between employer ocpupatipnal 
ratings and post-test scores in reading and speech* It would appear that 

these verbal skills ate related to job success, (A pilot project^ 196 7), 

'\ , ' ■ , ■ ' ^ . ' ' • . 

. e> Basic .Academic .Skillsrr -Grades * . ' — ' 



11. 6Z) for Project STAY p^tic 



Hamburger (1965) found aj^reater (statistically significant) improve- 
ment in class averages for a g::pup of co-op students when cpmjpared with a 
.control group. The. co-op grou]) did start-:from a lower baseline, however 
(69.97 to 73.53 .for the co-op ^md 70.69 to 71.95 for the control^) . 

Drane (1971) reported a-r(iduction in failing graces (ftom 13.1%-^to* 

.pants while Davis (1972) found that average 
grades went up for his sample ;n=15) from "D'* average ,to. "C+" from the 
Tlrst seme^ster to tHe second. / Project 'Outreach (1973) reported a 53^ decrease 
in •^D"-and "F" grades ovet a bne-y.ear period for ^ group of students (N=25) 
participajting in a drop-out prevention program, ' 

Kilbane and Plemirtg (19f72) repotted an incre.ase in^gracie-polnt average 
from .89 to 1.91 for boys 'ajld 1..57; .tjo' 2. 49 for girls in a drop-out prevention, 
program: for 9th graders. Ijd a statewide evaluation of a Cooperative Vocatidnal 
Educational fjcogram in the State of Illinois, it was found that the co-q6 
Student's, (Nf235) had an avprage gain of .502 in grade-point average vs, a drop- 
oi .066 f^ the 220 contrdls, (Schod^l, community^ and youth,. 1972). _ 

Sarthory (1971) was. the 'only one tb report a decrease in grade-point 
averages, His data wq^re jfxom 41 students In a summer \>6rk experience 
program. The* greatest decreas^e was^in 14 senior girls whidi ^ent from 
1.75 in the fall o£ 1969 to 1.13 one year later. ^ * ,* . 



'fi Measures of Attitudes and Alienation 

Lleubling and T robe (1965) found that' as a group program participants, tested 

^ wi th ^ the TAT (N-4 7) tended Xo: * \ 

' . Pi . 

1. Have difficulty in coping with reality, factors including low 

frustration tolerance, inability to postpone immediate gratifi- 
cation of needs, pessimism, and conflict between environmental 
demands and personal desires* 

2. Inadequate self*-awareness and confusion concerning s.ocial role 
requirements* • 

3. Impaired interpersonal relationships particularly with adults 
I and authority figures. 

4. Immaturity and dependency. ' 
5« Ambivalent and negative attitudes toward society^ ' */ 
It was concluded,- however, that these characteristics were reflections 

of experience and values of the sub-culture and were not pathological. 

Hamburger (1965) found .that the disadvantaged youth as a group 
started with a modest level of educational and vocational as.piration which 
did not change significantly during the study." (Hamburger, 1965, p. X3) 
These data were from the Rotter Imcomplete Sentence Blank. However,^ Drane 
(vl97l), working in two high schools in St. Louis,, showed that a drop-but 
prevention program (Project STAY) could be effective 4n improving. student 
aspirations as indicated by the "Student Progress, Chart" and the number of 
participants taking National College Entrance. tests. / 

With regard to self-concepts, Kaufman, et al. (1968a) reported an increas 
in feelings of s6if-esteem during; the course of an academically oriented 
dtop-out prevention program.. Attitudes were measured by the Gough Adjective 



3,8 



: chedt List, the Codpersml^h 5elf-Bsteem''ln^nt:pry and the Edwards Personal 
\ Preference Schedule/. Using an Adjecti^^ Check List self-concept Instrument, 



^ Jbsepji Olid Almca (1970) lrivestlgat!|d changes in s^lf-?attitudiBs before and 
after participation in a'^Vork Opportunity Center Prb^ram. -^i^one ^am^le^of 
54 students (average"age 18 yrs.'l mo.) who had been lenrolled over a period of 
16 weeks, approximately 66% of the group made improvements In self-perception 
scores. Another sample of HO students (average age .17.4 yrs.) who were ready to 
leave the prograi^ (to return to school, Start ^bs, cr graduate) showed a signi- 
cant increase in positive items. The males showe^ a jgreater increase in posi-r 
tive items than the females. 

Attitudes toward school and authority wete also me^ured using a 
/variety of instrument^. Using an attitude sqale' and obseiwations by 

consultants and evaluation specialists, Ericicson and Hamler (1972) reported 
. positive attitudes and lack J3f ^alienation toward school among 42 participants 
In & drop-out prevention program. These students were al*i in the program 
because they had Initially dropped out of school. . 

Drane (1971) found that after participation 1^ Project STAY, 84*4^ 
of the participants at one high schjool^and 81.5% at Ithe other had a positive 



attitude toward school as Measured by the Deinos Drop-out Scales. There were 
no baseline scores fbr comparison. In addition,, advisors percf^ived 80% 
of the students as having a positive attitude toward school. However, the 
goal of having, at least 50Z of the unemployed students participate in after 
school activities as an indication of positive . attitudes was not realized. 
Using an evaluator designed Semantic Differential Attitude Survey, Project . 
Outreach (1973) reported imptovement in students* attitudes toward adult 
authority and to their en^fironment. Davis (1972) found that after on^ year 



■ .. ^ 



ERIC 



44 



of participation in .a drop^-out prevention^program (N»223) 93% felt their atti 
tude toward school Improved "at least a little/' Most of the participants 
. felt that having "york expetietice" gave them an increased feeling of responsi 
bill ty at home (99%) , at school (97Z) , and at work (lOOZ) *. 

And post^attitude measures on program participants ^'(ZAS ninth 
graders) were token In three areas^"self as student," "spcial^spects of 
school""and "p^^^eption of teachers." Only negligible changes were reported 
in the boys' responses and these reflected a decline in positive feelings. 
The girls' Responses, however, reflected an increase in. positive feelings 

(Kilbane and Fleming, 1972). , ; ' 

\ ' I' . .\\' ' w ' ' ' ' ' 

Sarthory '(\t971) reported on a summer en5>loyment program involving work 

exploration, remedial education, and counseling foXyTl students. He found 

that attitude toward teachers improved and school appeared more desirable 

and Interesting but less useful at the end of the program.. The latter is 

nerlmps due to the discrepancy between what went' on in school and the real 

expectancies pf^mployers. , . 

\" - g. Acquisition of Social Skills - " 

Although the social requirements of work .are a major contribution to 

* job success or failure (Clark, et al. , X969), virtually nothing has* been 



done to evaluate this area, ^e one study Ippated^inM^e literature found 
^that, for those students who were unsuccessful in xet^^ning employment once 
placed in a job, i^eports from their employers li|dicated that the cause tei|ded 
to be lack of desired work attitude^ rather than lack of technical skills 
^ (Joseph .and Almeny 1970) . 



4b 



Po8t:.Br6gf am Objectlv^^ Itenge Goals ; ' - \ 

' a> Contlnued^Eaucatlen and .Employiaeiit If ' * . / • 

O ^ ' * ' - ' ' 

Ericksoh ac^JIamler (1972), .reporting on. the Benjamin iPranklin llrbah 

, Leagiie^Street Ac.adeiiijr In New /York, City, Jfound that pf the students from 

the 19 70-*"71, class upoti whom data were avail^le (N*58) were wording, or . in 

school. Of the 1971-72 class, ? 4% of the 110 students w^re still dh school . 

or working, ' They identified colTtiinuiiig one's education as a key evaluation. 

tool* It was Uyp6Lliesized that continuing participation in an educational 

progrxim reflected a 'self-confidence to achieve in a school setting, and 

as such would be a behavioral index of such self-confidence,. It was found that 

72%. of the 1971-72 sample (N«rl36) did in fact remain in school, ' 

Joseph and Almen (1970) found that of those who attended the Work 

Opportunity Cfenter, had earned an .average pf 4.9 academic credits each 

while a patched group of non-attendants made no academic progress.. Of ^-260 

respondents to follow-up questionnairet^, in Project STAY (St. Louis) 36.5% 

' * - ' , ' ' ' ' ..... ' ' " 

were eipployed and 47.3% were in post-high school educational progf&ms, , 
Of those surveyed' from die Work Study portion of the program (N«35), 28,6% were 
employed and 51.4% were in educational programs. Of those employed, however, 
\ only 30% were inoccupations related to the Work Study Training (Drane, 1971). . 
I Weber (1972), in a follow-up study of 41 st'uden^ graduates from a ^" 
"continuation high schoorMn the San Fr^cisco Bay area^^'f bund that" 60% 
' of the boys wer^ employed full-time, orie-third„.df the boys, were ienrolled ^ ^ 
in trade schools or ^apprentice programs, and' only two boys were unemployed. 
"Twp-thlrds of the/ girls ytexe, married;" they cons'tituteci the^ group/'npt 
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.employed full-tioe,"' (Weber, 1972, 573) ♦ In two. years of project on 
. upgr^diiig the employabLlity of drop-outs (N«3285, ages 16-20)> -McCarthy (1970) 
reported that 751 C23%) achieved full-time employment and 714 22!t) were 
enrolled in a regular full-time school or a training program, >. 

Over a 45-mon,th period with their Work Opportunity Center^ in Mliuieapblis , 
Joseph and Almen. (1970) found that 1250 of the 2761 enirallees .(45%) w^e 
placed in jobs* In a follow-up study (elapsed time unspecified) of 236 
students placed in jobs "about 60Z appear to have made* an Adequate adjustment 
at their work stations* • •Another 30Z had moved oh to another lob or had left 
the city* The remaining were seeking new jobs or were unemployed." (Joseph, 
and Almen, 1970^ p» 80) » They also found in their fallow-up stu^y that 
those who attended the Work Opportunity Center earned an average ok $^15 per 
hoyr more than a matched group of non-attendees (i,75/hr* vi^. i»60/hi^O 

Hombostel,. et al» Cno date) xeported on a study comparing three, 
groups of drop-outs with a control .group* The three drop-^out groups each 
had different training: academic only; academic and vocational; .and vocational 
only* They rej^orted no significant differences among any of the groups with 
regard to job satisfaction, employer ratings, proportion continuing in. their 
educatlonlor training, or in the extent of their general satisfaction ^^ith 
life» Over the two year follow-up they did fliid that .the Academic-Vocational 
and Vocational group fe^les were significantly higher than ,the controls with 
regard to length of tenure of employment arid average earnings > 
' ' b.#.: Citizenship, and Community^ Coricem W * 

, .When compared with the control group, Walthe?: (1967) reported that 

Black female .participants in*- four urban Neighborhoodjtouth Corps Programs 

' - ' . . ' \ ' 

were mo^e frequently eii^loyed and self-supportitsjg. There was a significant 

iti police cpntratts at two* of the sites.. The males showed slgiilficant 



lii9»rovemeht associated with continued education, both ' voca^^^^ 
acadenlic* One possible explanation for the favorable findings with respect, 
to Biack females, is that they . " . . I are easier tpwork with than are males',' • . 
they -have, more interest' in improving fchemselves.i and they have a more 
difficult time obtaining job traii^ing or jdfe placement without assistance." 
(Walther, 1967, pj 4). Sarthory (1971) also found that ..the summer work 
experience program, successfully reduced the crime rate and police encounters 
among participants. * - ... 

Erickson and Hamler (1972) examined "Trust," Communication and Concern^ 
With Community" as an index of self-development. While there were no 
quantitative data available it was' felt that th6 drop-out prevention 
program did "...reinforce the development of consciousness and concern 
for the community of the students." (p. 14). ' * . 

/c^' Other Long Range Effects 

^ ^- , 

Corazzini, et ai. (1966) examined the benefits and costs of vocational 

. - .' - ■ ■ . ' " ' ■ ■ « 

education ;in Wordiester, Massachusetts. . He concluded that the votfational 

school was not impressive as a drop-out prevention oschanism. The vocational 

sqhoql graduates were not favqred. over regular high school graduates by • 
/local firms and althdugh the vocational school aided graduates in job placement^, 

their career paths inside the firm Were the same as those of regular high school 
•graduates, further, the vocatioftil schools did not aid intra-regional movement j^^. 

(mobility) of workers'. The only .positive outcome appeared to l^i "inter-gene'StLnfi^^'' 

movement cJf workers (upward mobility).* However, this will \depend on whethe^lthe ^ ' • 

• '■• ■ ' ■ ' "\ ■ - ■■■,#* ■ \ • • • ■ ■ - • 

students actually follow the , career paths they, h^e 'chosen. ' . 
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SUMMARY AND CONCLUSIONS 



An extensive review of the literature has revealed little in the way 
of evaluation approaches,, techniques, or measures that are not ciirrently 
incorporated or accounted for in the existing evaluation system iesigned 
by Gibboney and Associates. The objective of the current modest effort 
was. to attempt to /identify crltei:ion*'referenced measures appropriate for 
the .assessment of /both the immediate and long range objectives 6f the CIC/CIP 
program. While v^e were able to identify several similar programs "jLn the 
literature, ,^ther^ have been few ,;4ttempts to systematically evaluate the outcomes 
of these programs 'to the extent currently being attempted by Gibboney and 
As.sociates« For the most part, the descriptions of the evaluation plans 
if described a D ^11 tended to be superficial. In some cases where there were 
attempts at devising a more comprehensive in-jiepth and long term evaluation, 
the data preseAited in the report itself were sketchy. It was difficult to 
tell froiii the reports, what was done in the way of evaluation, let alone what 
the outcomes were. In summary, the literature in this area is juite sparse 
'with little quantitative data. , . - 

With regard to the ineasuifes .themselves, since there w^s little reported , . 



oh criterion-referenced measures, this report was broadened to include all 



the measures identified in tife J.iteratture that might prove useful. These 
/ ran the gamut from standardized tedts to .program specific tools and from 

objective IQ tests to projective personality measures. There were., basically 
two general types of measures used. There were thpse measures which reported 
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actually observable behavior and those which were tests or indirect measures.. 
Tfie directly observable behavior measures included Items like attendance^ holding>, 
a job, not beconiing involved with the police^^ etc. It was of course assumed that 
the inciirect measures were related to and precursor^ of behavioral change. 
However, there was no real evidence presented that this Was^indeed the case. 

Unfortunately, few of the researcliers attempted to use measures employed 
by others^ so it is difficult to draw any conclusions regarding their general 
-Utility or applicability, (If a test is iBcd in only one program and no 
change is recor ded, ^ does this mean the measure is insensitive or the program 
is not working?) The one type of test tised most frequently was the standardized 
achievement test — particularly for reading and math. The Stanford, Iowa and" 

California Tests were the most common* In addition, another commonly used 

'* V, • - , ^ ' 

set of measures were those used to periodically ass^ess self-concept. These ^. 
measures incliided adjective check lists, sentence completion forms, and 
student self-descriptor forjns. In addition, structured interview forms werM 
used by counscLorp for these purposes as well, either as a substitute or in addi- 
tion to the self-evaluation, ^ ' ' ' ^ »• , 

Other scales and check lists were Used to assess attitudes toward authority, 
work, and school. It was thought that changes on these attitudes migW 
*be precurso^^of program success since they appear to be highly related tp-^ 
continuing in school and. to successful employment. As the QlC/CtB staff 
pointed out during one of the site visits, a major problfem appears to be ^ 
getting the student "turned around." Several of the programs inclu4ed 
personality tests, assuming that there would be some stability and hostility, etc. 
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^l!hey also iiried to obtain from these measures aa indication of. delayed grati- 

^ ' . . * ^ \ ' ' ' \ ' ' * 

ficatibn, like willingness to postponk monetary and other kinds of rewards 
while, completing one's education and training. The attitude or pef sodality 
measures also reflected thf. degree of feeling of control over bne*s future ' 

. . r 

•and destiny (the degree to which one feels he has something to say about ^ 
.what happens to him as, opposed to the idea that everything that happens is 
predetermined by where he is and who he is). Also, personality measures, have 
been used as Indicators of motivation, ambition, idesire to achieVd, aspiratior 
level, and desire for responsibility.' However, there appeared to .be nothing 
re.ported in the literature which wouJ.d aid in. determining the maturity^^^itlS: 
which career decisions are mad^. ^ 

Behavioral measures included the obvious thitigs like the percentage of 
sttidents who , stay In school, number of encounters with police, grades.'^ 
absenteeism, etc' The? criterion-referenced measures of program ^uctess were 
tied directly to the major objectives of the program. These included:^ ^'f^^ 
(1) keepiiig the student in school, (2) having the student achieve a^, l^as^ 
GKD or ^hlgli school diploma, and (3) having the student entet additional 
training or au entry level position for 'vhich he has been trained. in the: ^ 
program, or (4) having the student enter college.^ As' a long tange Fil^ilow-, 
up me.asure there were indices of job holding skills and "job success" as. 
measured primarily by intei^iew forms 6nd rating scalfss*^ 

There were few measures, however, directed at assessing the basic skills 
required for job application, such as what Xo do on^ an inte!rvi.e^, how to 
write a letter of ao-blication, how "to talk on the telephone and make aSx ^ 



appointment, etc. This despite obvious importance of these skills to 

' • * • . ' . , . \ * • . • ' ' * , 

^dbfaining and holding a jdb," " - ' / - 

The fact that so many different me^as.ures were used wouJLdxseem to' imply 

. > ' ' ' ' ' " It ' ' - • 

that no one was really satisfied with ahy given set of evaluation techniques. 

It Wouid appe^ar .that the data gathered iflth these, measures were not providing 

adequate infonriatlpn for the assessment of program eff ectdjjveness/ The mea^res 

fpund in the literature were for the most part relap^d to the specific . 

short-term expectations of each particular program. They were bashed on immediat 

program' outcomes and. were not emijirically tied to ^the ultimate long ran^e goals. 

Thus, thr-idata obtained consisted of many individual scores and indices which 

presented only an Incomplete and fragmented picture of program progress an§ \ 

success. It,., is apparent that what^may be desirable is an entirely new approach 

• . ' " '-^ • ' - . ' 

. ' / ^ ' . . , 

to program evaluation haUed on the answer to the basic question "Wiat have we 

. •.- ■ ■ • • , ' '-'v ■ ' . 

really done for this Individjial?" Is the individual really better off for 
haviijg been a product of the program? S.uch an assessment is neither summative 

nor formative but rather is oriented toward Impact and ultimate payoff It 

' ' \ ^ ' • ^ . . / . ' ■ . 

is not, asking the quq^tion of whether the 6bjectives>6f the program have been 

met but rather if' the individual participant has improved his or her pc teritial 

for a better life, i.e., were they' the correct objectives? \ 

- Xhis approach to evalua^tion has at least one serious' djawbapk. It is 

often difficult to relate any 4.ong term change in the individual's potential 

with the direct outcome of the* program itself. However;^ the advantage to 

■ ' ■ ■ • "1 --^ ■ ■' ■ • ' 

such^ an valuation approaches that the evaluation becomes much more meaningful 

in, terms of real and tangible benefitsr t^^ the/indivljdual. ~ There is virtually 1 

^ ■ . ^. . ' I ' ' ' . . ' ' 

iJothing in fhe literature having to do with this kind of approach' to assessment 



There are no precedents 



Ho ineasurea, and virtually no conceptual framework 



within wl 

4 

hnvv wo 



to, operate. So while the quostloit appears s 
Linproved tliiJ life of this human being — the. approa 



such a c} lestion is indeed' complex* 



Lmple— how 
z\\ to answering 
o 

centered arounfd 



Such ah assessment af>p^ach *as we. are describing is 
the Individual, ^d is crit^irlon-ref erenced and socially orignted^^^^^n 
addition, the measures should ideally be program independent ^ i.e. , 
independent of the stated program objectives. For^ example, the objectives 
stated in the program are having this ind^idual obtalii a high school diploma 
of a GED, obtain hands-on experience in several occupational areas, obtain 
an entry level job or training position, etc. The question still remains, 
however; That'is,^i'| the\ students obtain any or all of the stated objectives 

they better off than they would have been' If they had not 
; and are they better off than other people with similar 



of the program^, are 
attendeid the program 



backgrounds who have/ jnot\ attended the program? The iiiq)act assessmeh^^is 



I' 



based on a critical^ analysis of the .assumptions underlying the program aivd 
the validity of its jbjectives* It attempts to determine if the accomplishment 



of® the objectives contributes to the improved welfare individual. 

1 i ■• I- ' ■ i 

This aspect of ^'the evaluatjioh then takes the form of examining the 
an J obj€ 



program independently! and objectively from the 



outside. It s'tarts with the 



epd goal. and works back toward' the program,| itself . The first step will need , 
to be the est'ablishment of some criteria by which the success of the program 
can be assessed over a long range period. These aame^long range measured can al£ 



be applied at intermediate stages ^ro determine what progress Is being made. 
In addition, ibttermediate measures need to be identified which are predictive 

of the: ultimate criterion. These may include some of the more traditiojnal 

'l ' . \ ^ 
tests and measures^^ identified and •some new measures developed specifically 



for this purposes For example, orye ^jossiblii measure for. assessing the ultimate 
, imp^ict of ^he progranf may be* the general concept of "fnyestment." In an AIR 
impact evaluation conducted overseas (Krug and Juifg> 1974), it. wa% prd^sed ' 
^ that one measure, of the iiwlividual^s '^^ngagement" -with society (as opposed to 
alienation from) is the amount of time, money and effort he jLs willing to give to 
his community • this may^be reflecteci i^yplunteer work, charity contributions, 
political activity, .etc. Using the same measure one caif assess an individt^al^s 
willingness to invest time,** money and/or effort in his own continued self-/ 
development, e.g., continued education, travel, parjjicipation in cultural 
events, etc. ' - 

^ . J ' ■ ■- ' . . ■ . 

ill addition, interim progress measures need to be, ^developed which e^e 
related t6 these impact measured, ^or ^example, one .ppsaible progress measure^^^-^ — ' 
related to ultimate program success may be a relat^ionship between geographic 
mobility in job-seeing behavior and maturity and self-concept. It fflj»y bie^ tjia*-^ 
the more mature, self-confident individual^ \^i.ll be willing and able to seek 
jobs or employment outside of his normal sphere of operation. This allows for 

, , ' ^ ' ^ V / ~ ' ^ * 

greater choice anc^^ better probability of fiildthg suitable employmegt. 

""-^^ ' " ' ^ ' . ' . ' ' , ^' * 

In the CIP Program being run by the .OlCs of America, the^ ultimate goal ^1^^' 

is both unambiguouj3 and obvidi>^-to improve the social and economic well- ^ 

being of those -individuals participating iri the. prog£,am. Any measures of program 

siicceiss must center around the individuals s^ocial ^nd economic development. 

The inteflilror progress measure3 should reflect the growth, or lack of growth," 

. ■ . '■ V ^ ■ ' , ' . :, ' . ■-. . * 

exhibited by the indiVi'<}p^l from the time he enters the program, to some pre- 

' . / - ' ^ --^^ - . " 

selected end'poipt. Itj is» suggestexi that thi^ period of evaluation may well * 

. ' : , \ . ^ ' . • ' ' ' • ' 

extend for many years, in, the future. To fulfil ghis .aspect of the evaluation 



^. 0 ^ 



/ ^ • 



a considerable fejEfort .needs to be Undertaken .to devel9p and validate l)oth the ul- 



timate and Intermediate progress raeasurib. Such ^n effort j^ould rtbt only benefit 
iho current program, but would .contribute immeasurably to 'the advanceb^ent of. ' 



^ ' tthe' state-of-the-rart in evaluation, 

' / / of 
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APPENDIX-'A;-. 



MEASURES 



A. Inte-illRence 



Culture Fair Intelligence -Test, Scale 2 
Forias A- &• B ■ / " 

* Lorge Thbmdike Intelligence Test 



Otis Tntelltgerice Test, Gamma Form E . 

Ol'Ls Qurck-Scorlng Mental Ability, 
Forms KM & FM 

Wesclrler Adult Intelligence Scale 

B. Aptitude 

• ^ ^ . \ ^' 

Aracd Forces Qualifying Test 5 

3 [ Airman Qualifying Examination 

Bennett Mechanical Comprehension Test, 
^' Form-AA 

tjCiyil Servipe Examination, 

' DtEfcrentJal Aptitude test 
' General Aptitude Test Battery 



- -C. General Achievement and Skill Measures 
Agency Intern Evaluation Form 

Bojrei Reading Inveritory 

it 

Career Development 'Achievenfent Test 



•REFERfeNCES' 



(Kaufman et al. , 1968a) 

(A pilot prpject--6ccupational 
training, 1967; Beecher and 
DiPasquale,* 1962)' . 

(Hamburger, 1965) 

(leubiing and Trobe, 1968a) ; 

(Leub ling. and Trbbe, 1965) 



(Savitzsky et aX/; 1965) 
(SAvitzky et al.,/1965)/ 
(Sharar et al., 1969) 



(Exemplary vocational education 
prog tam--Finai Report, 1973bj 
^Savitzky et al*,, 1965) 

(BeeS&er and DiB^squale, 1962; 
JIamburger, 19^.5) ' 

(Dayton," 1973; ExeraplarPvocational 
ec^cation programs-Interim Report?, 
19)33.; Hornbbstel et al. , ho datej, 
SaviMky^ et al. ,' 1965; Thurston 
and Weber, 1971) . 



i(6f f-campus experience. . . for high 
schoolers, 1972) . 

"'*''«» 
(Fripst and .Pilgrim, 1969) ^ 

(Randolph and Holmes, eds., 1972), 
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sv.Ctites Vocational ^Development Inventory 
Cumulative Record • 



(Randolph and Hplme^, eds. , 1972) 

(Erlck^onanT Hamlet,, 1972; - 
- Kaplani^ 1967) ^ ^l. . 



The. Diagnostic Reading Test > 

** 

Differential -^Aptitude Test, Battery ^ 

Dolch Basic Slg4ift Word Text 

Every. Pupil Scholarship test in Typing 
I & n " ' 'v' ^ 

Gates Macginltie Reading Test 



Gilmore Oral Reading , ^ 

Goertzel Job Success Rating Scale, 
. Form -A 

Grade*. Point Average 



Heltt Simplified Shorthand Test 

Iowa. Aclvieyement Tests < 

Job Corps Reading and Kath Placement 
Test3 

Jdb Ratings 

Kottmyer Diagnostic .Test of Word ,^ 
Perception Skills. 

McCall-Crabs Standard Test 

Melleribruch Office Skills 



. (Career College, 1969) 
' (Hamburger, 1965) 

. (Frost and Pillgrim, 1969) 

■V 

(Hornbostel et al^i' no date) 



(Decker, anc* Anderson, 1969;. 
Frost and Pilgrim,' 1969; Sha^rar 
et al., 1969) , ' . ' 

(Frost* ^d -Pilgrim, 1969) 

(Hornbostel et al., ho date) * 

(Iphahce and Sai:thory, 1972; Dayis, 
1972; Evaluation of i:he ih-school ^ 
youth work-training project,* 1968; , 
LoWe-et%i^, 1970 ;.^P^ject Outreach, 
19731 Royst^, 1970; Sarthory, 1971) 

(Horhbostfei. et. al. , ^no date) ' - 

(Project Outreach, 1973) 

(Twelfth month evaluation—Project 
Interchange,^ 196,8) I 

(Off-rcampus experiences., .^or 
hig;h schoolers, 1972) 

(Frost ancl Pilgrim, 1969) 

(Frost and Pilgrim, \l969) 
'{Hornbostel et al»,' no da^e)' 



Metropolitan. Achievement Tests 

Morrisoh^McCalfi Spelliflg. Scale 
IjhBW York University ..Speaking Test 

O'Cdnn or linger and Tweezer Dexterity^ 
^ Test' / . • 

Office Ma^Jiine Mechanical, Jest ' 

Beafeody Picture Vocabulary Test 

R^urdue Tests-Welding, Shfeetmetal Work, « 
Mkchin:£sts an^ Kachine Operation 

Questionnaire Survey 

Scienfi^l- Research Associates Phonic Survey 

sequential Tests of Educational Progress, 
Eroms*3A atid 3B-. 



Stanford Achievement Test 

- ' - \ ■ • . . 

t 

f 

• - • ' ' ' ' 

Teacher Progress Reports 

Vocational Training* Ratings by Instruction 

Personality '^ - . ' ; • ■ 

Galifarnia Psychological Inventory ^ 

California Test of Fersonaltty, 
' rForms AA and BB ' , 

: ' • ^ \ ' " * 

Case Study J . . 



Citizenship *Grade Point Average * *. 
Counselor-Evaluation ^, ^ 
■Edwards Personal {'reference Schedul'e 



(Bitter, 1966;; Sharar et aX,,» ; 
1969; Tweifth month evaluation- 
Project Interchange, 1963) 

*' • ♦ • . ■ . ■ " 

(Frost aild Pngrlm, 1969) 

^(A pilot project — occupational 
training^ 1967) - - 

(Leubling and Trpbe,' 1965) ' 



(Hornbdstel etal.^ no date)^ 
(Joseph. and Alroen, 1970) 
(Hombostel et aL«, no, date) 

(Weber, E., 1972) 

(Frost and Pilgrim, 1969) 

(Hombostel et^.al, , no date) 



(Beecher an4^DlSi$5uale,^ .1962; 
Ericksoh and Haml^i:, 1972;' Frost 4nd 
Pilgrim, 1969 ; Hamburger, '1967; \ . / 
Kaplan, 1967-; Kkufman et al, , .l96'8a) 



(Hombostel ^et al.., no date^ 
Kaplan,, 1967) ' . 



(Sharar et al. ,'1969)^ 



'.(Thurston and Weber," 1971),, 

, (A pilot prbject--occupational^ . 
trainlngy 1967; tiorpibost^l et al*,, 

i969> - ■ \ • ^ // • : 

(A pilot: projectr-pdcupatlonal 
-training,' 1967; S^vltzky et'al. , 1965)* 

(Yon^r,. 1967) .'^ ' . - 

; "* ■ ■ •• ' 

(EricKSOn -and Hawlet, 1972) , . /. ' .. 
\ (Kaufman, 196'8a)^^ 



Gough Adjective Checklist . 

" iV?K£ kaxiety Scale Queistionnaire 

Potential Drop-out Instrument (PDL) 

Rotter Incomplete Sentences Blank, .High 
' • . School Form 

* Self-BirectedVsearch Specimen Set 

Sel'f-Esteem Inventory 

Sociometric Form ^ *^ ' 

' ^ ^ ^ ' s ■ • ' ' • 

Mi* A ttit udeg and Interests 

' • » -t • * 

Attitude Survey and Observation 



(Kaiufman, 1968a) 



Brayfielci-Iiothe Jobc Satisfaction 
Questionnaire ' 

Business arfd" Industry Job Survey 



► Occupational Adjustment Rating Scale 
Ohio Vocational Interest Survey ^^v 



Perceptions of Career ^Education 
Questionnaire Survey 



I * 



(Hombo^stel *et al. , no dat4^ 
(Randolph: and Holmes;. edsV,\l972> 



Rural-Urban Orientation inventory. 
'Semantic Differential" • * 
Skills Program vs.. Diploma Program 

' Social.Cl^s Value Orientation 

* ... 

§,taf f Reactipif IPorm, 

Stydfent Opinionnaire for Career » 
\, Education Perceptions " 

• Teacher Observation 



.(Hamburger, 1965) 



(Kortzek, 1972) 



(Randolph arid Holmes, eds., 19^)' 
(Rudolph and HblmeSy eds., 19fe) 



(Lowe, 1970; PrdjectvOutreach, .1^73) 
(Hombostel et al., no\date) , 



(East and Dolab, 1968b) 

(A pilot project — occupational 
training., 1967) 

(Dayton, 1973; Exemplary Wqationa] 
education—Interim , 19 7 3a ^^o rizek ,1 
1972) ' \ " . ^ ' \ ■ 

(Korizek, 1972) 

(Connau'tej- Job' Corps — Final Report, I \ " 
1970; »aYls„ 1972; Dayton, hl3i ^ - \ 
.East and*Dolan, 1968a'^ ErickLon .'^ * 

r ' a^id Hamier, 1972;; Hombostel \et al, 
' * rib date; Sarthoxy.,' *197l) 

' . . ^ N - . ^, • 

(Hombostel et ai.^ n'o d^te) 
(Kaplan, 1967)^ . ' ' ' 
(Kaufmah et al.*, '1968a) ' : 

(Hornbostel et al. , no dat^) 
(Kaplan, 19^7). ' 



(Korizek,,lB72) 



Y.J (Randolph and. Holmes eds., 1972; * > 
I Thuifs ton ind Weber, i9 71)' • ' 
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Vocational Sentetice Completion Blank 
WelngaVttMi Picture Inventory 
The Wafik Values Inventory (Super) 
California Study Methods Survey 

Case, Study 

Demos Drpp-out Rating Scale 
Internal-External Control Scale 
Injtertriews\ 



Kuder Preference Record,' Form, A 



Kuder Preference Redord, Form DD - . 
Life jp fanning Questionnaire 
Milwaukee Student Survey ^ . . . 
Minnesota Vocational Interests Inventory 

F'. Other Measures 

Hamburger Revised Occupational Scale 
. - V for Measuring Socio-=Economic Status 

Keystone^ Visual Survey 

Medical -.Report Tallies * - 



(Cangemi'^ 1964) 

CSharar et al.,* 1969) 

(Hamburger^ 1975) . % 

(A Pilot project-- occupational — — 
training, 1967) 



(Savitzky et ai^ 1965) 

^(Randolph and Holmes, eds., 1972), 

(Career College, 1969) 

(East and Ddlan, 1968a; Erickson 
andHamler, 1972; Exemplary 
vocational education program — 
Final .Import, 1973b; Hamburger^ 
1965; Hornbostel et al., no date; 
. Kaplan, 1967; Twelfth month evalua- 
• tion — Project Inter.change , 1968) 

(Beecher.and DiPasquale, 1962; 
Exemplary vocational education — 
Interim Report, 1973a; .Hornbostel . 
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Yunker, 1967) 

(Korizek, 197^2) 

(Hamburger, 1^65) ,\ 

(Career Development Program, 1972) 

(K^^T;lfman et al. , 19 6 8a j Sharar 
et al. , 1969). 



(Hamburger, 1965; Leubling and 
Troije, 1965) 

(Frost and Pilpim, 1969) 
(Kaplan, 1967).. 
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Polidc Encounters 



Pupil Attendance Record . 



Pdpil Holding Power Datav Instrument 
Retention Rate (% stayingi in school) 



Reduction of Crime Rate 

ScliDOl Drop-out Research In\terview 
S:hedule 



(Chance an<J'Sarthory, 197Z) 

"^(A pilot project— ^occupational 
training, U967; Chance and 
Sarthory, 1972; 6avis, 1972; 
Hamburger, 1965; Kaplan, 1967; 
Project .Outreach, 1973; Roystoii, 
1970; Yunker, i967) 

(Gangemi^ 1964) 

(Chance and Sarthory, 1972;, 
Erickson and Hamler, 1972; 
Project Outreach, 1973) \ ' , 

(Sarthory, 1971) 

(Hornbostel et al., no date) . 



